-
Notifications
You must be signed in to change notification settings - Fork 228
Workflows Streaming - Add transport via datachannel for guaranteed frame processing #1704
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Run a test with this small test script and the following terminal command (you need to reference a video on your machine): Command: Script: |
|
not ready yet |
|
bugbot run |
hansent
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
hansent
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Description
This PR adds a new way of providing frames for our webrtc streaming implemention by creating a new datachannel where we can send extracted frames from a recorded video, and are guaranteed to have them all processed in the provided quality.
Update, implemented binary chunk streaming for frame ingestion and sending back data via the datachannel. both directions:
Corresponding app PR: https://github.com/roboflow/roboflow/pull/8537
Overview
Both directions (client → server and server → client) use the same binary chunking protocol with a 12-byte header followed by payload data. This ensures:
Binary Message Format
Header Fields:
Payload Types:
Type of change
Please delete options that are not relevant.
How has this change been tested, please provide a testcase or example of how you tested the change?
See comment below
Any specific deployment considerations
For example, documentation changes, usability, usage/costs, secrets, etc.
Docs
Note
Implements a binary chunked data-channel protocol for client→server frame ingestion and server→client responses, with a new
use_data_channel_framesmode enabling data-only processing without a media track.create_chunked_binary_message,parse_chunked_binary_message,ChunkReassembler,send_chunked_data(48KB chunks).send_chunked_data(replaces plain text.send).use_data_channel_framesflag inWebRTCWorkerRequestand processing classes.upstream_framesdata channel: reassemble JPEG chunks, decode with OpenCV, enqueue frames for processing.process_frames_data_only()when using data-channel frames and no video track.VideoFrameProcessor/VideoTransformTrackWithLoopacceptuse_data_channel_frames; manage internal frame queue and received frame IDs.data_outputandstream_outputadjustments via control channel.Written by Cursor Bugbot for commit d7e3295. This will update automatically on new commits. Configure here.